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ABSTRACT 

^^""""^-^^^he components of an educational program, designed to 
ensure the attaintit^nt of specific objectives, are described^ Program 
objectives included rhe introduction of a new individualized 
instructional system opfevj9.ting within the same constraints 
experienced by public schooi^taff , insurance that each child in the 
program would receive his fai^fcsshare of instruction, and insurance 
that children would be validly mtaasured on criterion and norm 
referenced tests • The ef fectivenes^^sgf the program's components in 
attaining these objectives is discuss>84^ Observations concerning the 
efficiency of performance contracting as^*»a^rocedure for bringing 
dbout change also are presented • (Author) 
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AN ANALYSIS OF A GUARANTEED ACHIEVEMENT PROGRAM TO ACHIEVE 
SPECIFIC EDUCATIONAL OBJECTIVES, OP^ PERFORMANCE CONTRACTING, 

CHICAGO STYLE 



When we in the Chicago Public Schools embarked on a performance 
contract program, it was with the intent of designing a program to 
achieve certain specific goals within the constraints of our large 
city school system. This paper is an attempt to offer jud,gments 
concerning the effectiveness of the program in meeting the desired 
goals . 

The program was designed (a) to improve the reading skills of 
innercity pupils, (b) to avoid the pitfalls reported in early per- 
formance contract programs, (c) to be one which could be continued 
by the public schools, and (d) to be satisfactory to various Interes 
ted groups, including community residents, the general pub 1 £c^ as.,^^^^--^ 
represented by the Boa rd of — Edurcation, personnel at several admini- 
strative levels, teachers and the Chicago Teachers Union, and the 
funding Model Cities Agency. 

The last goal stated above was attained to a large degree 
through the creation of a Performance Contract Management Commit tee . 
This committee, which was appointed by the General Superintendent of 
Schools and received his continuing support, functioned particularly 
during the planning and proposal review stages, but it has continued 
to function throughout the life of the performance con t r act ' program • 
Participation of the community was enhanced because the project was 
. funded by the Chicago Model Cities Agency which includes mechanisms 
for community participation in planning and review. The Performance 

1. 



Contract Management Committee, chaired by the Associate Superinten- 
dent for Educational Program Planning, consisted of tx^o board 
members, community representatives from each of the four Model Cities 
Target Areas, a Model Cities staff representative, a teachers* union 
representative, two principals and two district superintendents of 
schools in which the program was to take place, the three area asso- 
ciate superintendents, and several central office staff such as the 
administrator of Board of Education Model Cities Programs and the 
Director of Research. Although the committee was large enough to 
have been potentially unv/ieldy, in practice this was not the case, 
partly because much important work such as prep^r^^i^oi^^ Request 
For Proposals (RFP) and detailed reviewi^-^^' proposals was done by 
subcommittees. Most important, this comprehensive committee has 
achieved agreement between the various groups represented on what 
could hc;ve been a very controversial program. 

The first three goals stated above are recast in terms of the 
objectives which appear below. Each statement of an objective will 
be followed by a description of those provisions of the program which 
were designed to achieve the objective, which will be followed in 
turn by an attempt to describe the success of these program provisions 

Objective ; To introduce a new instructional system, with financial 
risk to be assumed by an independent contractor, which (a) would 
operate within the same constraints as those experienced by public 
school staff and (b ) could be "turnkeyed" to the schools. 

^Although some of the data contained in the sections which follow 
\^have not been previously reported , most of the results are drawn 
from the '^Final Evaluation-Chicago Model Cities Performance Con- 
tracting Reading Project** prepared by the Institute for the Develop- 
ment of Educational Auditing (I.D.E.A.), the evaluation/auditing 
contractor retained for this program. 



To achieve this objective a performance contract was entered 
into with Learning Research Associates (LRA) of New York to teach 
reading for one hour per day to approximately 1800 children in thir' 
teen "Centers for Accelerated Learning" located in ten Model Cities 
Target Area schools. The awarding of the contract was preceeded by 
solicitation and evaluation of proposals received as the result of 
Request For Proposals (RFP) , followed by lengthy negotiation with 
the contractor. The evaluation of proposals by the committee focus 
on the extent to which the instructional system was individualized, 
cost -^-F-Ft^tp-k^vp , npf n%rp r^1-y.- H pp Pn H Pn f on expe nsive equj,pTnent or 
remodeling, and on an estimate of its potential for continued accep 
tance and use by classroom teachers. The contract provided for pay 
ment to the contractor to be based only on the reading achievement 
of children as measured by norm-referenced and criterion-referenced 
tests, while making the. contractor responsible for virtually all 
instructional expenses associated with the program. In addition, 
the contractor was required to (a) utilize public school teachers 
from the schools as his on-site staff following Board of Education 
and union-agreed-upon-pro cedures (b) work with all children in a - 
classroom, and (c) through inse tvice . and on-the-job training enable 
the teaching staff to develop the expertise to continue the 
instructional program. 

The evidence clearly indicates that "a new instructional 
system" has been introduced into the ten schools. The program was 
closely monitored by a representative of the evaluation/auditing 
contractor who spent one day in each center every thirteen days. 



The reports which were produced clearly indicated that the center 
teachers utilized the individualized LRA reading system almost ex- 
clusively in instruction in the reading centers and that during the 
course of the year most of the classroom teachers who came with their 
classes to the centers also became proficient in their use of the LRA 
individualized system. Indeed, according to the reports, most of the 

pupils became proficient in many aspects of the individualized system 

/ 

as well* 

Further evidence that "a new instructional system" has been 

introduced is offered by an analysis of pupil activities utilrzing 

2 

structured observations in the centers. The results of these obser- 
vations are presented in Table 1 and, for ' comparison purposes, the 
results of observations in all classrooms in seven other target area 
schools, which themselves had been attempting to move toward greater 
individualization of instruction, are presented in Tables 2 and 3. 
While the reader may find many of the observations of interest, the 
most direct comparison of the observations in the performance contract 
and comparison classrooms is to be found between the last column of 
Table 1 and the first column of Table 3. Particularly noteworthy are 
the more than 5,0% of pupi Is engaged in "Independent Involvement with 
Materials " in the performance contract schools compared to about 24% 
in the other schools and the 6% of pupils engaged in "Teacher-Directed 
Group Instruction-Two or More" in the performance contract schools 

compared to about 44% in the other schools. Of further interest Is 
2 

The observation form used was adapted by the research and evaluation 
staff from a form devised by Lindvall, Yeager, Wang, and Wood inclu- 
ded in "A Comparative Investigation of the Efficiency of Two Class- 
room Observational Methods" by Mary Ann Kissel, a doctoral disserta- 
V tion. University of Pittsburgh, 1970. Copies of the adapted form and 
lists of activities to be included in each category are available from 
the author of this paper upon request. 
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the fact that in the performance contract centers while the pupil 
activity category of "Teacher-Directed Group Instruction-Two or More" 
contains about 13% of the primary pupil observations, it contains 
less than 2% of the Intermediate pupil observations. 

The phrase "with financial risk to be assumed by an independent 
contractor" embodied the notion that if the program were "successful" 
the contractor would get full pay, if "unsuccessful" h ewoiU-d'-ge t 
nothing whileje«i.nj_3ll..^«pe,^^ 

U^n^tdrateTT^^ment scale, with some most-alle individual children 
>eing expected to gain 2.5 grade equivalents and master as many as 50 
reading objectives in a year's instruction. Complete lack of success, 
as implied in the payment scale, was that children would do no better 
than an estimate of their previous achievement, for instance that some 

less-able children would ga-fr. -^^r r - i . . 

6 mv,rc Ciian .0 ^rade equivalents or 

would master fewer than 10 reading objectives in a year of Instruction. 

The program as funded by the Model Cities Agency, was supplemen- 
tary to the regular expenses in the public schools, and the contractor 
could only receive payment based on the test gains which, as seen 
above, would be greater than those which would have been' otherwise 
expected of the children in the program. The maximum to be received 
per child was $223.00 and the maximum to be received for all children 
was $400,000.00. m return thl'Tont ractor guaranteed to pay for (a) 
the salaries of the fifteen public school teachers utilised full time 
in. the program, (b) most of the furniture and equipment utilized, (c) 
^the materials, and (d) the costs of training teachers in the use of 
^t^e instructional system, other than teacher pay for "inservice" over- 
time. Preliminary calculations indicate that the contractor's out-of- 



pocket expenses in Clii :af o have approximated $265, 000,00, while the 
amount earned on test scores approximates $150,000.00 and tlie amount 
which the Board of Education is committed to pay for furniture, 
equipment, and materials approximates $50,000.00. This financial 
plight of the contractor has come about in some degree because of 
very optimistic bidding on hoped-for achievement of children. 

Because payment was calcuJ^a^i_4oreach child in the program and 
because most children did achieve in the expected range on criterTon*^^ 
referenced tests, the contractor did earn the approximate $150,000.00 
based on testing, even though the mean standardized achievement test 
scores were in the lower part of the range bid for individuals and 
were not appreciably different from test score means in control 
schools. In my judgment, determining the success of a program solely 
on first year test results probably offers an unne ccess ari ly severe 
measure, and does not take into consideration the almost inevitable 
start-up problems. The question of whether the program is "successful 
can be better answered after a second year of operation. In any case 
the performance contract as agreed upon has resulted in the instruc- 
tional contractor assuming a large part of the financial risk. 

^ The desire to have the performance contractor "operate within the 
same constraints as those experienced by the public school staff" was 
directly related to the desire to be able to have the program con- 
tinue after the expiration of the contract. In general the program 
has operated under the general public school constraints, except 
that materials were more expeditiously secured at the onset by the 
contractor than they could have been by the schools, which was very 
helpful. In a larger sense the supplemental nature of the funding 



violates the notion of "operating witliin constraints" and, as 
support is withdrawn for reasons unrelated to the success of the 
program, this may prove fatal. Operating within the constraints has 
helped to insure a high degree of teacher morale and acceptance, 
which is admittedly somewhat higher for center teachers than for 
classroom teachers. 

Whether the program can be ef f ec t ively " turnkey ed" is current- 
ly being tested. As long as supplementary funding is maintained it 
is apparent that the public schools can, and are, maintaining about 
the same program in the second year which the contractor instituted 
in the first year. If supplementary funds completely disappear, 
the main residuals will probably be the skills and insights of 
individual teachers in schools which will be hard pressed to come up 
with the necessary funds for the relatively expensive instructional 
management system and materials, and which will face problems of 
shared--use or dismantling of the attractive and well equipped read- 
ing centers. 

Objective ; To insure that each child included in the program would 
recieye his "fair share" of individualized instruction. 

To achieve this objective a payment scale based on the achieve- 
ment of each child as an individual was negotiated. This scale was 
also graduated, so that the child who was most able had to achieve 
more than a less able child for the contractor to achieve full payment 
Furthermore, a portion of the payment was to be held for a year after 
the conclusion of the program to insure that children retained any 
benefits received. Further, the probability that the objective 
would be met was thought to be enhanced by providing that center 



teachers and field consultants would be Chicago Public School teachers. 

There is no indication that any child or group of children 
systematically received attention to the exclusion of any other 
children in the program and there is considerable evidence which 
indicates that children did receive individual attention according 
to their needs. This evidence consists in part of ol * vations on 
and reports by those in the centers, but the mosu ..mpelling evidence 
is the record of objectives mastered by individual children. In the 
intermediate centers there was no correlation be tween the number of 
objectives mastered by children, which averaged approximately thirty 
during the year, and the gains achieved on the standardized tests. 
This is probably not only related to the fact that lower level 
objectives are less time consuming, but also to a practice of 
center teachers of tailoring the prescription for each child so that 
he or she would be mastering an objective almost weekly. 

In retrospect, some of the provisions for sliding payment 
scales for individual children seem to have been more elaborate than 
necessary to insure attainment of this objective. As with any pre- 
vention system, it is difficult to estimate effectiveness in the 
absence of that which is to be prevented. Conceivably the eff.ect 
could have occurred by the inclusion of such provisions - in the RFP, 
To my knowledge, there has been no hint, and very little possibility, 
of LRA altering instruction to influence payments. 

Objective ; To insure that pupils' progress in reading was validly 
measured through the use of (a) criterion referenced and (b) norm 
referenced tests. 



To achieve this objective the testing program incorporated 
the following features: 

(a) criterion referenced testing was based on the day to day 
diagnostic and check test program built into the instruc- 
tional system, with the auditing contractor randomly 
retesting children on the attainment of objectives in 
each center on a continuing basis 

(b) two standardized reading tests were administered to each 
child at reading level rather than at grade level on a 
pre-posttest basis. 

It became apparent in the planning stages that attempting to 
administer an all inclusive criterion referenced pre and posttest, 
or alternately, to tailor a criterion referenced pre and posttest 
test for each child, offered insurmountable logistical problems. 
At the same time i^: was realized that criterion referenced pre and 
posttests were ijnbedded in the instructional system. These tests 
were utilized with randomly sampled children being rechecked by the 
evaluation auditing contractor, with penalties being assessed 
against criterion referenced test payments for discrepancies greater 
than ten percent between the findings of the evaluator and the 
center teacher. Seven centers achieved the 90 percent or better 
agreement , with i:he remaining 6 achieving between 80 and 89 percent 
agreement . Feedback from the evaluator seemed to be particularly 
helpful in bringing greater agreement as the program progressed. 
The tests described were individually administered and usually con- 
^sisted of ten or fewer items, with mastery defined as 80 percent correct. 



The above information speaks nore to the reliability than to 
the validity of the criterion referenced tests. The pre, post, and 
check tests for the 230 objectives generally appeared to have face 
validity, and were certified as such by the audi ting /evaluation 
contractor. There was no evidence from the instructional contractor 
that the tests had ever been submitted to any empirical test analysis 
other than continued use in the program. Under such circumstances 
the user is left with fewer assurances concerning validity, and 
reliability, than would be possible with more rigorous test construc- 
tion . 

The method of administration of the standardized achievement 
tests seems to have insured that pupils in the program were validly 
measured, particularly in the intermediate grades. In order to 
test at reading level rather than at grade level, children were 
retested using a different level of the test if they scored fewer 
than 25 percent or more than 90 percent of the item^: correctly. 
When this correct level of the test was determined for the Metropoli- 
tan Reading Test, 1970 Edition, the Iowa Test of Basic Skills Reading 
Test was administered at the indicated level. Agreed-upon procedures 
called for using, where possible, only those test scores in the 25 
to 90 percent range, and where possible, utilizing t'lie mean of scores 
from the two tests. Two other provisions helped to insure that test 
results would be valid indices of achievement. The auditing evalua- 
tion contractor examined all instructional materials to certify that 
they did not "teach to the test", and tests were to be administered bj 
the auditing evaluation contractor. The pretests were administered 
^y the contractors • personnel with classroom teachers present. Due 
to some reports. of "non-standard" test administration by some teacher?, 
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by mutual agreement the posttests were administered by classroom 
teack-ers with monitors being supplied by the audi ting /evaluation 
contractor. Because the instructional contractor challenged the 
posttesting conditions, the results of twelve testing sessions, out 
of approximately two hundred, were not used. The twelve cases con- 
sisted either of disruptive pupil behavior or of improper administra- 
tion such as incorrect time limits. It seems fair to say that this 
testing was subjected to much closer scrutiny than would be the case 
in ordinary testing in the schools. Achievement test results are 
shox^n in Tables 4 and 5, on the basis of which the following conclu- 
sions are offered: 

There were no appreciable differences between the 
achievement and test gains of the project and control 
pupils. (ANCOVA, not shown, performed on intermediate 
scores did show statistically greater gains at the .05 
level in three of the nine tests shown for intermediates.) 

Although the use of two tests offers reassurance in the 
present program, additional information offered is not 
such as to recommend the use of two tests in future 
programs. 



Testing at reading level seems to have insured that the 
children were accurately measured, and any additional ' 
problems with administration were accepted by both 
teachers and pupils. Probably as a result of- testing 

reading level, it is interesting to note that reported 
fifth grade gains are greater than fourth grade gains and 
sixth grade gains are higher still, a pattern that has not 
O often been found in studies on inner city pupils. 



TABLE 4 



Posttest scores of Primary 1 pupils on Metropolitan 
Reading Achievement Tests, 19 70 Edition. 



School 



Pro.i e ct 



Con trol 



A 
J 
H 

B 
E 

N 

0 



^ 1.6 



H 1.6 



-4 1 



1.7 



Hi. 6 



1 1.7 

Wl.6 



Mean MAT Grade-Equivalent Scores 



TABLE 5 

Cracla GciuLvalcnl; gains of in terr.odl ata puijils. 

4th GRADE 



1.0 
.8- 

MEAN .6-- 
GAIN 
SCORE .4-- 

.2-- 



40 



•1= .49 



M V C 
Project 



.66 



48 't^ 



M V C 
Control 
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1.0 
.8-- 

MEAK' .6-- 
GAIN 
SCORE .4" 

.2-- 



5th GRADE 



.75 



56 .56 



M V C 
Project 



.56 



.69 

T .60 



M V C 
Control 



ERiC 





1.0-- 




.8-- 


MEAN 


.6 - 


GAIN 




SCORE 


.4 -- 




,2- 



6th GRADE 



77 



.66 



.45 



M V C 
Project 



87 



.60 '^J 



M V C 
Control 



*M « Metropolitan total Reading Achievement Test Scores, 1970 Edition 
V - Vocabulary scores on Iowa Test of Basic Skills 
C = Comprehension scores on Iowa Test of Basic Skills — 
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Final Observa tions 

The performance contract program was demonstrably successful in 
introducing a new individualized reading system into the Chicago 
schools. The individualized reading system has been accepted and is 
being used daily by teachers and pupils, VJhether this system will 
be more effective than other methods cannot be answered on the basis 
of the results of achievement testing to date. Despite start-up 
p7oblems, which made for less than a full year of actual experience 
with the new system, the children in the project did abot. as well as 
those in the control schools. Based on these considerations, it was 
decided to continue the instructional program, but not the performance 
contract, for another year in .order to determine the effectiveness of 
the reading system. 

Based on the Chicago experience, several advantages of the per- 
formance contract process may be identified: 

(a) The Board of Education and staff are very interested 
in establishing a critical and thorough evaluation 

which will determine the value of a performance contract - 
perhaps more than with some, "in-house" programs • 

(b) The low price paid for services received reflects 
the competitive bidding, 

(c) Th^ independent contractor brings an urgency to the 
process of educational change which tends to be 
understood by the staff. 
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These advantages must be balanced af.ainst some rea] costs: 

(a) The usual RFP-selection-negotlai:lon procedure for 
securing a contractor is quite t;ime consuming if 
done thoroughly, as are the calculations, and 
sometimes negotiations, V7hich ara necessary to 
determine the payment. If a management consul- 
tant contractor is employed some, but by no means 
all, of these costs will be made explicit. 

(b) The economies of scale which would be available 
if the school district engaged in many such 
contracts are not generally available. 

(c) Teachers may be resistant to full participation 
because of antagonism to the contractors' 
profit motive . 

An alternative to performance contracting which retains some of 
the advantages while reducing some of the costs is available. This 
is to retain a contractor to introduce a new system, as the result 
of an .RFP and selection process, without making payment to him con- 
tingent on pupil achiev-ement. The contractor's pay would still be 
contingent on the performance of his agreed-upon services, and-if 
evaluation is carefully built into the program his ability to stay 
in business could well depend on the outcome. Furthermore, some 
portion of the staff energies which would go into payment scale 
negotiations could be redirected beneficially toward helping 
recipient schools to be receptive initially. With equal interest 
^^nd support, this alternative may well provide as high or higher 
level of performance without a performance contract. 



